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EXECUTIVE  SUMMARY 

This  report  describes  in  detail  the  procedures,  methods,  and  resources  Parsons  used  to  complete 
the  demonstration  project  at  the  fonner  Pole  Mountain  Target  and  Maneuver  Area  (PMTMA)  for 
Environmental  Security  Technology  Certification  Program  (ESTCP)  Project  MR-201157 
(Demonstration  of  MetalMapper  Static  Data  Acquisition  and  Data  Analysis).  The  2011  ESTCP 
Unexploded  Ordnance  Classification  Study,  Pole  Mountain  Target  and  Maneuver  Area, 
Wyoming  was  conducted  with  two  primary  objectives: 

>  Test  and  validate  detection  and  discrimination  capabilities  of  currently  available  and 
emerging  advanced  electromagnetic  induction  sensors  developed  specifically  for 
discrimination  on  real  sites  under  operational  conditions. 

>  Investigate  in  cooperation  with  regulators  and  program  managers  how  classification 
technologies  can  be  implemented  in  munitions  and  explosives  of  concern  (MEC) 
cleanup  operations. 

Parsons’  responsibilities  on  this  project  included  only  the  processing  and  analysis  of 
MetalMapper  data  collected  at  the  PMTMA  site  by  Sky  Research,  Inc  (Sky).  The  MetalMapper 
is  an  advanced  electromagnetic  induction  system  developed  by  Geometries,  Inc.,  with  support 
from  the  ESTCP.  It  has  three  mutually  orthogonal  transmit  loops  in  the  Z,  Y,  and  X  directions 
and  contains  seven  tri-axial  receiver  antennas  inside  the  Z  (bottom)  loop,  allowing  21 
independent  measurements  of  the  transient  secondary  magnetic  field.  Data  were  collected  over 
target  locations  statically,  such  that  one  data  point  was  collected  for  each  target  selected  for 
investigation.  Sky  personnel  collected  MetalMapper  data  over  2,370  targets  at  the  site,  pre- 
processed  the  data,  and  submitted  background  corrected  .CSV  files  for  each  target  to  ESTCP, 
who  then  forwarded  the  files  to  Parsons. 

The  2,370  data  files  were  inverted  and  analyzed  using  the  UX-Analyze  add-on  to  Geosoft’s 
Oasis  Montaj  software  package.  Once  analysis  was  complete,  theoretical  ranked  dig  lists 
(theoretical  because  all  targets  were  intrusively  investigated  regardless  of  the  indicated  stop  dig 
points)  were  submitted  for  scoring  by  the  Institute  for  Defense  Analyses.  Dig  list  scoring  was 
based  on  the  number  of  targets  of  interest  (TOI)  correctly  identified  as  items  that  should  be  dug 
and  the  number  of  non-TOI  or  clutter  items  that  were  correctly  classified  as  items  that  did  not 
need  to  be  intrusively  investigated.  Dig  lists  submitted  by  Parsons  were  scored  against  the 
ground  truth  data  generated  during  the  intrusive  investigation  performed  at  the  site  following 
MetalMapper  data  collection.  Parsons  submitted  two  ranked  dig  lists  for  this  project.  Both  dig 
lists  correctly  identified  all  of  the  TOI  on  site  as  TOI,  so  the  only  distinction  between  the  two 
was  the  amount  of  clutter  correctly  classified  as  clutter.  The  better  performing  of  the  lists 
reduced  the  amount  of  clutter  that  needed  to  be  intrusively  investigated  by  approximately  79 
percent,  and  the  other  reduced  the  clutter  digs  by  approximately  73  percent. 

Parsons  became  familiar  with  UX-Analyze  software  used  for  processing  and  analyzing  the 
MetalMapper  data  on  data  from  two  previous  projects.  There  had  been  no  major  changes  to  the 
software  between  projects,  and  no  implementation  issues  were  noted  during  processing. 
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1.0  INTRODUCTION 


Currently  up  to  90  percent  of  excavation  costs  on  most  unexploded  ordnance  (UXO)  /  munitions 
and  explosives  of  concern  (MEC)  projects  are  related  to  removing  scrap  metal  that  does  not 
represent  an  explosive  hazard.  Significant  cost  savings  could  be  achieved  through  the  use  of 
geophysical  discrimination  methods  that  could  reduce  the  number  of  excavations  required  to 
remove  explosive  hazards  from  sites.  The  objective  of  this  project  is  to  demonstrate  the  use  of 
advanced  electromagnetic  induction  (EMI)  sensors  in  static  data  acquisition  mode  and  associated 
analysis  software.  To  achieve  these  objectives,  a  controlled  test  was  conducted  at  the  former 
Pole  Mountain  Target  and  Maneuver  Area  (PMTMA). 

1.1  BACKGROUND 

The  Fiscal  Year  2006  defense  appropriation  contained  funding  for  the  “Development  of 
Advanced,  Sophisticated  Discrimination  Technologies  for  Unexploded  Ordnance  Cleanup.”  The 
Environmental  Security  Technology  Certification  Program  (ESTCP)  responded  by  conducting 
the  UXO  discrimination  study  at  the  former  Camp  Sibert,  Alabama.  The  results  of  this  first 
demonstration  were  very  encouraging.  The  conditions  for  discrimination  were  favorable  at  this 
site  and  included  a  single  target-of-interest  (4.2-inch  [in.]  mortar)  and  benign  topography  and 
geology.  All  of  the  classification  approaches  demonstrated  correctly  identified  a  sizable  fraction 
of  the  anomalies  as  arising  from  nonhazardous  items  that  could  be  safely  left  in  the  ground.  Both 
commercial  and  advanced  sensors  produced  very  good  results.  Camp  San  Luis  Obispo  (SLO), 
California,  was  the  site  for  the  second  study,  which  provided  greater  challenges  in  topography 
and  a  wider  mix  of  targets  of  interest  (TOI).  Again,  the  results  were  very  positive,  with 
increased  discrimination  of  TOI  versus  nonhazardous  items.  In  2010  and  early  2011,  the  third, 
fourth,  and  fifth  ESTCP  studies  were  conducted  at  the  fonner  Camp  Butner,  North  Carolina,  the 
former  Mare  Island  Naval  Shipyard,  California,  and  fonner  Camp  Beale,  California,  which  all 
included  smaller  TOI.  Great  success  was  achieved  identifying  37-millimeter  (mm)  projectiles, 
fuzes,  and  larger  TOI  at  Camp  Butner  and  Camp  Beale  with  the  advanced  sensors.  Mare  Island 
results  are  currently  unavailable. 

To  build  upon  the  success  of  earlier  studies,  ESTCP  sponsored  a  sixth  study  in  2011  at  a  site 
with  a  wide  range  of  TOI  and  variable  terrain.  The  selected  range  at  the  fonner  PMTMA 
included  approximately  50  acres  of  open  field  potentially  contaminated  with  ordnance  ranging  in 
size  from  37mm  to  3in.  projectiles  and  mortars. 

1.2  OBJECTIVES  OF  THE  DEMONSTRATION 

This  type  of  approach  has  the  potential  to  reduce  the  number  of  excavations  required  to 
effectively  remove  the  explosive  safety  risk  (MEC)  at  a  given  site,  which  would  result  in 
significant  cost  savings  related  to  the  closure  of  fonnerly  used  defense  sites.  The  cost  savings  are 
expected  to  be  particularly  significant  at  removal  action  sites.  Parsons  is  currently  involved  with 
U.S.  Army  Corps  of  Engineers  on  several  MEC  and  recovered  chemical  warfare  material  sites 
that  could  be  used  for  additional  testing  and  refining  of  the  process  required  for  this  type  of 
discrimination  approach. 
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This  demonstration  consisted  of  the  cued  interrogation  of  previously  identified  EM61-MK2 
anomalies  using  the  MetalMapper  advanced  EMI  sensor.  The  objective  of  the  MetalMapper  data 
collection  and  analysis  was  to  accurately  classify  each  of  the  targets  as  either  a  TOI,  likely 
representing  UXO,  or  non-TOI,  representing  another  type  of  anomaly  source  such  as  UXO 
fragments  or  other  metallic  clutter.  Specific  performance  objectives  were  developed  for  the 
classification  study,  including  the  selection  of  a  dig/no-dig  threshold  that  retained  all  of  the  TOI 
targets  while  minimizing  the  number  of  false  alarms,  minimizing  the  number  of  targets  classified 
as  “can’t  analyze,”  and  estimating  target  parameters  such  as  location  and  depth  correctly. 

1.3  REGULATORY  DRIVERS 

As  part  of  the  cleanup  of  former  Department  of  Defense  (DoD)  sites,  buy-in  is  required  from 
regulatory  agencies  at  the  federal,  state,  and  local  levels.  The  advancement  in  classification 
sensors  and  their  successful  deployment  at  real-world  sites  needs  to  be  documented  for  their  use 
to  be  accepted  by  the  applicable  regulatory  agencies.  Their  acceptance  of  the  use  of  this 
technology  at  sites  for  which  they  are  ultimately  responsible  will  be  particularly  important  with 
the  potential  for  DoD  budget  cuts  to  affect  the  amount  of  money  that  will  be  available  for  future 
remedial  actions. 
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2.0  TECHNOLOGY 


2.1  TECHNOLOGY  DESCRIPTION 

The  MetalMapper  is  an  advanced  EMI  system  developed  by  Geometries,  Inc.,  with  support  from 
the  ESTCP.  The  MetalMapper  draws  elements  of  its  design  from  advanced  systems  currently 
being  developed  by  G&G  Sciences,  Inc.  (supported  by  Naval  Sea  Systems  Command,  the 
Strategic  Environmental  Research  and  Development  Program  [SERDP],  and  ESTCP),  and  by  the 
Lawrence  Berkeley  National  Laboratory  (LBNL)  with  support  from  SERDP  and  ESTCP.  It  has 
three  mutually  orthogonal  transmit  loops  in  the  Z,  Y,  and  X  directions  and  contains  seven  tri- 
axial  receiver  antennas  inside  the  Z  (bottom)  loop.  Typically,  the  transmit  loops  are  driven  with 
a  classical  bipolar  pulse-type  time  domain  electromagnetic  wavefonn  (i.e.,  alternating  pulse 
polarity  with  a  50%  duty  cycle).  Depending  on  the  survey  mode  (e.g.,  Static/Dynamic),  the 
fundamental  frequency  of  transmission  can  be  varied  over  the  range  1 . 1  l<f  <810  hertz  (Hz).  The 
seven  receiver  antennas  allow  2 1  independent  measurements  of  the  transient  secondary  magnetic 
field. 


The  data  acquisition  computer  (DAQ)  is  built  around  a  commercially  available  product  from 
National  Instruments.  The  National  Instruments  DAQ  is  a  full-featured  PC  running  Windows  7. 
The  DAQ,  electromagnetic  transmitter,  and  batteries  for  the  system  are  packaged  in  an  aluminum 
case  that  can  be  mounted  on  a  pack  frame,  on  a  separate  cart  such  as  a  hand  truck,  or  on  the 
survey  vehicle  such  a  tractor.  The  instrumentation  package  also  includes  two  external  modules 
that  provide  real-time  kinematic  global  positioning  system  location  and  platform  attitude  (i.e., 
magnetic  heading,  pitch,  and  roll)  data.  These  modules  are  connected  to  the  DAQ  through  serial 
RS232C  ports.  A  block  diagram  of  the  DAQ  system  is  shown  in  Figure  2-1. 


Figure  2-1:  DAQ  and  DAQ  Functional  Block  Diagram 


A)  MM  DAQ  assembly  with  Panasonic  terminal. 


B)  Function  block  diagram  of  MM  system 


The  MetalMapper  has  two  modes  of  data  collection,  dynamic  and  static.  Data  collected  in 
dynamic  mode  result  in  data  files  containing  many  data  samples.  Generally  speaking,  dynamic 
mode  data  are  collected  while  the  antenna  platform  is  in  motion.  Static  mode  data  collection  is 
employed  for  cued  surveys.  As  its  name  implies,  the  antenna  platform  remains  static  or 
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motionless  during  the  period  of  data  acquisition.  Depending  on  the  acquisition  parameters  (e.g., 
sample  period  and  stacking  parameter)  it  can  take  tens  of  seconds  to  complete  a  static 
measurement.  The  results  of  the  static  measurement  are  written  into  a  binary  data  file  containing 
only  a  single  data  point  representing  the  average  (stacked)  result,  usually  over  tens  or  even 
hundreds  of  repetitions  of  the  transmitter’s  base  frequency. 

Data  are  acquired  in  time  blocks  that  consist  of  a  fixed  number  of  transmitter  cycle  “repeats”. 
Both  the  period  and  the  repeat  factor  are  operator  selectable  and  are  varied  in  multiplicative 
factors  of  3.  The  MetalMapper  also  averages  an  operator-specified  number  of  acquisition  blocks 
(NStacks)  together  before  the  acquired  data  are  saved  to  disk.  The  decay  transients  received 
during  the  off  times  are  stacked  (averaged)  with  appropriate  sign  changes  for  positive  and 
negative  half  cycles.  The  decays  in  an  individual  acquisition  block  are  stacked,  and  the  decays  in 
that  block  are  averaged  with  other  acquisition  blocks  (assuming  the  operator  has  selected  NStack 
greater  than  1).  The  resultant  data  are  saved  as  a  data  point.  A  photo  of  the  typical  configuration 
of  the  instrument  used  for  collecting  cued  data  is  shown  in  Figure  2-2. 


Figure  2-2:  Antenna  Array  and  Typical  Deployment  of  the  MetalMapper  System 


In  its  present  (third  generation)  form,  the  MetalMapper  technology  has  been  demonstrated  and 
scored  at  the  Standardized  UXO  Technology  Demonstration  sites  at  Yuma  Proving  Grounds 
(Blind  Grid  only),  at  Aberdeen  Proving  Grounds  (Blind  Grid  plus  Direct  Fire  and  Indirect  Fire 
Areas),  and  most  recently  at  SLO,  Camp  Butner,  and  Camp  Beale  in  connection  with  2009 
through  2011  Classification  Studies  carried  out  by  ESTCP.  The  performance  of  the 
MetalMapper  at  these  sites  is  documented  in  formal  reports  issued  by  the  Aberdeen  Test  Center 
and  by  the  various  demonstrators  who  analyzed  the  data  collected  at  SLO,  Camp  Butner,  and 
Camp  Beale. 
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2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

There  are  a  few  advanced  EMI  sensors  that  are  similar  to  the  MetalMapper  in  theory  and  design, 
with  the  most  comparable  being  the  Naval  Research  Laboratory’s  TEMTADS  5x5  and  the 
LBNL  Berkeley  UXO  Discriminator  (BUD).  The  TEMTADS  5x5  consists  of  25  pairs  of 
transmit/receive  coils  oriented  in  a  5x5  grid  pattern,  approximately  2  meters  (m)  to  a  side.  The 
BUD  is  composed  of  three  orthogonal  transmitters  and  eight  pairs  of  differenced  receivers. 
These  instruments  have  been  part  of  the  ongoing  ESTCP  Classification  Demonstrations,  and 
similar  results  have  been  documented  for  all  three  during  previous  projects.  The  main  advantage 
of  the  MetalMapper  is  that  it  is  currently  commercially  available,  while  the  other  two  are 
generally  only  used  by  the  organizations  that  developed  them  and  are  very  limited  in  number. 

The  greatest  limitation  of  the  MetalMapper  is  its  size,  both  of  the  sensor  itself  and  of  the 
accompanying  computer,  screen,  and  cables.  The  system  is  designed  primarily  for  use  in 
relatively  flat,  open  fields  and  cannot  currently  be  used  effectively  in  wooded  areas. 
Additionally,  its  effectiveness  in  areas  with  extremely  high  anomaly  densities  is  currently 
unknown,  although  data  were  collected  in  high  anomaly  density  areas  during  a  demonstration 
study  at  performed  Fort  Sill  in  late  2011.  Results  for  this  study  are  currently  unavailable. 
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3.0  PERFORMANCE  OBJECTIVES 

The  primary  performance  objectives  for  this  demonstration  include: 

•  Evaluating  whether  classification  techniques  will  work  at  the  former  PMTMA  site; 

•  Evaluating  where  classification  techniques  will  work  at  fonner  PMTMA;  and 

•  Evaluating  the  cost  effectiveness  of  classification  techniques  in  the  areas  at  the  former 
PMTMA  where  classification  is  detennined  to  be  effective. 

The  specific  performance  objectives  for  this  demonstration  are  based  on  the  objectives  stated  in 
the  project  demonstration  plan  (Parsons,  2011)  and  are  summarized  in  Table  3-1. 


Table  3-1.  Performance  Objectives  for  this  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Maximize  correct 
classification  of 
targets  of  interest 
(TOIs) 

Number  of  TOIs 
retained 

•  Prioritized  anomaly 
lists 

•  Scoring  reports 
from  Institute  for 
Defense  Analyses 
(IDA) 

Approach  correctly 
classifies  all  TOIs 

Maximize  correct 
classification  of  non- 
TOI 

Number  of  false 
alarms  eliminated 

•  Prioritized  anomaly 
lists 

•  Scoring  reports 
from  IDA 

Reduction  of  false 
alarms  by  >  50% 
while  retaining  all 
targets  of  interest 

Specification  of  no¬ 
dig  threshold 

Probability  of  correct 
classification  and 
number  of  false 
alarms  at 
demonstrator 
operating  point 

•  Demonstrator- 
specified  threshold 

•  Scoring  reports  from 
IDA 

Threshold  specified 
by  the  demonstrator  to 
achieve  criteria  above 

Minimize  number  of 
anomalies  that  cannot 
be  analyzed 

Number  of  anomalies 
that  must  be  classified 
as  “Unable  to 

Analyze” 

•  Demonstrator  target 
parameters 

Reliable  target 
parameters  can  be 
estimated  for  >  98% 
of  anomalies  on  each 
sensor’s  detection  list. 

Correct  estimation  of 
target  parameters 

Accuracy  of  estimated 
target  parameters 

•  Demonstrator  target 
parameters 

•  Results  of  intrusive 
investigation 

X,  Y  <15  cm  (la) 

Z  <10  cm  (la) 
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3.1  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOI 

One  of  the  two  main  objectives  of  this  demonstration  was  to  correctly  classify  all  seeded  items 
and  any  MEC  items  remaining  at  the  site  as  TOI. 

3.1.1  Metric 

The  metric  for  this  objective  was  the  number  of  items  on  the  MetalMapper  anomaly  list  correctly 
classified  as  TOI. 

3.1.2  Data  Requirements 

MetalMapper  data  were  analyzed  to  create  a  prioritized  dig  list,  which  assigned  each  target  to 
one  of  three  categories:  1)  TOI,  2)  non-TOI,  or  3)  can’t  analyze.  The  targets  classified  as  either 
TOI  or  “can’t  analyze”  were  considered  “dig”  targets.  The  Institute  for  Defense  Analyses  (IDA) 
used  scoring  algorithms  to  compare  the  “dig”  targets  to  the  list  of  items  identified  as  TOI  during 
the  intrusive  survey. 

3.1.3  Success  Criteria 

The  objective  is  considered  met  if  all  items  of  interest  are  correctly  labeled  as  TOI  on  the 
prioritized  anomaly  list. 

3.2  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-TOI 

This  is  the  second  of  the  two  primary  measures  of  the  effectiveness  of  the  classification 
approach.  In  addition  to  correctly  classifying  TOI,  the  effectiveness  of  the  MetalMapper  in 
discriminating  munitions  is  a  function  of  the  degree  to  which  responses  that  do  not  correspond  to 
TOI  can  be  eliminated  from  consideration  during  the  intrusive  investigation. 

3.2.1  Metric 

The  metric  for  this  objective  was  the  number  of  targets  on  the  ranked  anomaly  list  created  using 
the  MetalMapper  data  that  were  correctly  classified  as  non-TOI. 

3.2.2  Data  Requirements 

MetalMapper  data  were  analyzed  to  create  a  prioritized  dig  list,  which  assigned  each  target  to 
one  of  three  categories:  1)  TOI,  2)  non-TOI,  or  3)  can’t  analyze.  The  targets  classified  as  non- 
TOI  were  considered  “no  dig”  targets.  IDA  used  scoring  algorithms  to  compare  the  “no  dig” 
targets  to  the  list  of  items  identified  as  non-TOI  during  the  intrusive  survey. 

3.2.3  Success  Criteria 

The  objective  is  considered  met  if  more  than  50%  of  the  non-TOI  items  can  be  correctly  labeled 
as  non-TOI. 

3.3  OBJECTIVE:  SPECIFICATION  OF  NO-DIG  THRESHOLD 

In  a  retrospective  analysis  as  performed  in  this  demonstration,  it  is  possible  to  tell  the  true 
classification  capabilities  of  a  classification  procedure  based  solely  on  the  ranked  anomaly  list 
submitted.  In  a  real-world  scenario,  all  targets  may  not  be  dug,  so  the  success  of  the  approach 
will  depend  on  the  ability  of  an  analyst  to  accurately  specify  its  dig/no-dig  threshold. 
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3.3.1  Metric 

The  probability  of  correct  classification  and  the  number  of  false  alarms  at  the  dig/no  dig 
threshold  in  the  prioritized  dig  list  are  the  metrics  for  this  objective. 

3.3.2  Data  Requirements 

Following  data  collection,  MetalMapper  data  were  analyzed  to  create  a  prioritized  dig  list,  which 
assigned  each  target  to  one  of  three  categories:  1)  TOI,  2)  non-TOI,  or  3)  can’t  analyze.  The 
category  into  which  each  target  is  placed  was  determined  using  a  decision  statistic  developed 
during  analysis  of  the  MetalMapper  data.  The  dig/no  dig  threshold  for  this  project  was  the 
decision  statistic  value  that  separates  targets  classified  as  TOI  from  those  classified  as  non-TOI. 
IDA  personnel  used  its  scoring  algorithms  to  assess  the  results. 

3.3.3  Success  Criteria 

The  objective  is  considered  met  if  more  than  50%  of  the  non-TOI  items  can  be  correctly  labeled 
as  non-TOI  while  retaining  all  of  the  TOI  at  the  specified  threshold. 

3.4  OBJECTIVE:  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE 
ANALYZED 

Anomalies  for  which  reliable  parameters  cannot  be  estimated  using  the  collected  MetalMapper 
data  cannot  be  classified.  These  anomalies  must  be  placed  in  the  dig  category,  which  reduces  the 
effectiveness  of  the  classification  process. 

3.4.1  Metric 

The  number  of  anomalies  for  which  reliable  parameters  cannot  be  estimated  is  the  metric  for  this 
objective. 

3.4.2  Data  Requirements 

Those  targets  for  which  parameters  could  not  be  reliably  estimated  were  identified  as  such  on  the 
prioritized  dig  list  submitted  following  analysis  of  the  MetalMapper  data. 

3.4.3  Success  Criteria 

The  objective  is  considered  met  if  reliable  parameters  can  be  estimated  for  >  98%  of  the  targets 
on  the  prioritized  dig  list. 

3.5  OBJECTIVE:  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

This  objective  involves  the  accuracy  of  the  target  parameters  that  are  estimated  in  the  first  phase 
of  the  analysis.  Successful  classification  is  only  possible  if  the  input  features  are  internally 
consistent.  The  obvious  way  to  satisfy  this  condition  is  to  estimate  the  various  target  parameters 
accurately. 

3.5.1  Metric 

Accuracy  of  estimation  of  target  parameters  is  the  metric  for  this  objective. 
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3.5.2  Data  Requirements 

The  inverted  or  fit  locations  detennined  for  each  target  during  MetalMapper  analysis  and  the 
locations  of  recovered  items,  as  recorded  by  the  intrusive  teams,  were  compared  to  determine  the 
difference  between  the  two. 

3.5.3  Success  Criteria 

The  objective  is  considered  to  be  met  if  the  estimated  X,  Y  locations  are  within  15  centimeters 
(cm  [la])  of  the  actual  locations  and  if  the  estimated  depths  are  within  10  cm  (la). 
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4.0  SITE  DESCRIPTION 

As  Parsons  personnel  were  did  not  perform  any  of  the  on-site  tasks  for  this  project,  limited  site 
information  is  available.  The  following  was  taken  from  the  ESTCP  Live  Site  Demonstrations 
Plan  for  the  former  PMTMA  (ESTCP,  2011).  The  former  PMTMA  is  a  62,448.15  acre  site 
located  in  Laramie,  WY.  The  demonstration  was  conducted  in  the  Bisbee  Hill  Maneuver  Area. 
An  aerial  photo  of  the  demonstration  area  is  shown  in  Figure  4-1. 

4.1  SITE  SELECTION 

This  site  was  chosen  as  the  next  in  a  series  of  sites  for  demonstration  of  the  classification 
process.  The  first  site  in  the  series,  fonner  Camp  Sibert  in  Alabama,  had  only  one  TOI  and  item 
“size”  was  an  effective  discriminant.  A  hillside  range  at  the  former  Camp  San  Luis  Obispo  in 
California  was  selected  for  the  second  of  these  demonstrations  because  of  the  wider  mix  of 
munitions,  including  60-mm,  81-mm,  and  4.2-in  mortars  and  2.36-in  rockets.  Three  additional 
munitions  types  were  discovered  during  the  course  of  the  demonstration.  The  third  site  chosen 
was  the  former  Camp  Butner  in  North  Carolina.  This  site  is  known  to  be  contaminated  with 
items  as  small  as  37-mm  projectiles,  adding  yet  another  layer  of  complexity  into  the  process.  The 
fourth  site,  the  former  Mare  Island  Naval  Shipyard  (MINS)  in  Vallejo,  CA,  was  selected  because 
of  an  opportunity  in  the  Navy’s  remediation  schedule  at  MINS  to  conduct  the  study  in  the  midst 
of  their  ongoing  munitions  response  project  and  prior  to  the  upcoming  removal  action  in  2012. 
The  fifth  site,  Camp  Beale  in  Yuba,  CA,  was  selected  for  demonstration  because  it  is  partially 
wooded  and  is  thought  to  contain  a  wide  mixture  of  munitions. 

The  demonstration  site  totaled  approximately  50  acres.  The  EM61  cart  surveyed  the  entire  site 
with  the  exception  of  the  fenced  area  in  the  upper  left  part  of  the  site.  The  demonstration  site  is 
shown  in  Figure  4-2.  This  site  was  selected  because  of  its  wide  mixture  of  munitions  and 
variable  terrain.  The  smallest  known  munition  on  the  site  is  the  37-mm  projectile,  the  largest 
known  items  are  3-inch  projectiles  and  mortars,  with  a  range  of  munition  sizes  in  between. 

4.2  SITE  HISTORY 

The  PMTMA  was  established  in  1879  as  the  Fort  D.A.  Russell  Wood  and  Water  Reserve.  The 
land  status  alternated  between  national  forest  and  military  reservation  from  1897  to  1925.  The 
Pole  Mountain  area  has  also  been  known  as  the  Crow  Creek  Forest  Reserve,  Fort  D.A.  Russell 
Target  and  Maneuver  Range,  Fort  Francis  E.  Warren  Target  and  Maneuver  Range,  Pole 
Mountain  Reservation,  Pole  Mountain  Training  Annex,  and  Warren  Training  Annex.  It  was 
extensively  used  before  1959  as  a  target  and  maneuver  area  by  the  Army,  the  Reserve  Officers’ 
Training  Corps,  the  Citizens’  Military  Training  Corps,  various  National  Guard  units,  and  the 
Department  of  the  Air  Force. 

4.3  SITE  GEOLOGY 

Site  specific  geologic  infonnation  was  not  included  in  the  ESTCP  demonstration  plan.  However, 
near  surface  geologic  conditions  were  not  expected  to  adversely  impact  the  functionality  of 
either  the  EM61-MK2  or  the  MetalMapper.  No  significant  geologic  effects  were  noticeable  in 
the  MetalMapper  data  collected  during  the  project. 
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Figure  4-1:  Location  of  Demonstration  Site 
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Figure  4-2:  Aerial  Photograph  of  the  Demonstration  Site 
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4.4  MUNITIONS  CONTAMINATION 

A  large  variety  of  munitions  have  been  reported  as  used  at  PMTMA.  Physical  evidence  for  the 
following  items  was  discovered  during  the  RI: 

•  Projectiles  containing  high  explosive  (HE)  filler  (37-mm  to  155-mm,  and  2.95-inch); 

•  Shrapnel  projectiles  (75 -mm  and  3 -inch); 

•  37-mm  projectiles  (inert  and  unfuzed) 

•  3 -inch  Stokes  mortars  (practice,  fuzed); 

•  60-rnm  mortars  containing  HE  filler;  and 

•  Small  arms  ammunition  (.30-caliber  and  .50-caliber) 


13 


5.0  TEST  DESIGN 


5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

The  objective  of  this  program  was  to  demonstrate  a  method  for  the  use  of  classification  in  the 
munitions  response  process.  The  three  key  components  of  this  method  are  1)  collection  of  high- 
quality  geophysical  data  and  principled  selection  of  anomalous  regions  in  those  data,  2)  analysis 
of  the  selected  anomalies  using  physics-based  models  to  extract  target  parameters  such  as  size, 
shape,  and  materials  properties,  and  3)  the  use  of  those  parameters  to  construct  a  ranked  anomaly 
list.  Each  of  these  components  was  handled  separately  in  this  program,  with  different  contractors 
responsible  for  different  tasks.  Parsons  was  only  responsible  for  the  processing  and  analysis  of 
data  collected  by  another  contractor.  Target  parameters  extracted  during  processing  were  passed 
through  classification  routines  that  were  used  to  produce  prioritized  anomaly  lists  ordered  from 
the  item  that  the  classification  routine  determined  was  most  likely  a  munition  through  the  item 
regarded  as  the  most  likely  to  be  nonhazardous. 

Validation  digging  was  also  coordinated  by  the  Program  office.  The  prioritized  anomaly  lists 
were  scored  by  the  IDA,  with  emphasis  on  the  number  of  items  correctly  labeled  nonhazardous 
while  correctly  labeling  all  TOIs. 

The  primary  objective  of  the  demonstration  was  to  assess  how  well  each  demonstrator  was  able 
to  order  its  ranked  anomaly  list  and  specify  the  threshold  separating  high-confidence  clutter  from 
all  other  items.  The  secondary  objective  was  to  detennine  the  classification  performance  that 
could  be  achieved  by  each  approach  through  a  retrospective  analysis. 

5.2  SITE  PREPARATION 

Parsons  was  not  involved  in  site  preparation  for  this  project.  Site  set-up  and  logistics  were 
performed  by  URS  Corporation,  and  details  regarding  this  aspect  of  the  project  should  be 
contained  in  their  report. 

5.3  SYSTEM  SPECIFICATION 

The  MetalMapper  sensor  and  data  acquisition  system  are  described  in  detail  in  Section  2.1.  All 
MetalMapper  data  for  this  project  were  collected  by  Sky  Research,  Inc.  (Sky).  Site-specific 
MetalMapper  configuration  should  be  discussed  in  detail  in  Sky’s  report. 

5.4  CALIBRATION  ACTIVITIES 

All  MetalMapper  data  for  this  project  were  collected  by  Sky.  Any  MetalMapper  calibration 
activities  performed  on  site  should  be  discussed  in  detail  in  Sky’s  report. 

5.5  DATA  COLLECTION  PROCEDURES 

All  MetalMapper  data  for  this  project  were  collected  by  Sky.  Specific  data  collection  activities 
performed  on  site  should  be  discussed  in  detail  in  Sky’s  report. 
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5.6  VALIDATION 

All  anomalies  on  the  master  list  were  excavated  by  a  team  led  by  the  URS  Corporation.  At  the 
conclusion  of  data  collection  activities,  all  anomalies  on  the  master  dig  list  were  excavated. 
Each  item  encountered  was  identified,  photographed,  its  depth  measured,  its  location  determined 
using  cm-level  GPS  (global  positioning  system),  and  the  item  removed  if  possible.  These  ground 
truth  data  were  used  for  evaluation  of  the  dig  lists  submitted  by  various  analysts. 
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6.0  DATA  ANALYSIS  AND  PRODUCTS 


The  MetalMapper  was  used  to  collect  static  data  over  2,370  targets  identified  at  the  fonner 
PMTMA  based  on  EM61-MK2  data.  The  processing  and  analysis  steps  that  were  used  to 
generate  a  dig/no  dig  decision  for  each  target  are  described  below. 

6.1  PREPROCESSING 

Raw  MetalMapper  data  are  collected  and  stored  as  .TEM  files.  The  MetalMapper  acquisition 
software  uses  a  convention  for  assigning  a  unique  name  to  each  data  file  without  the  need  to 
manually  enter  the  name.  The  operator  supplies  a  prefix  for  the  root  name  of  the  file  (e.g., 
“Static”).  The  acquisition  software  then  automatically  appends  a  five-character  numerical  index 
to  the  filename  prefix  to  form  a  unique  root  name  for  the  data  file  (e.g.,  StaticOOOOl).  The  index 
is  automatically  incremented  after  the  file  has  been  successfully  written.  Although  the  Target 
identification  (ID)  is  not  used  as  the  file  name  in  the  .TEM  file,  the  Target  ID  is  stored  in  the  file 
according  to  name  of  the  target  highlighted  on  the  MetalMapper  screen  during  collection. 

Pre-processing  of  the  .TEM  files  was  performed  by  Sky  personnel;  it  consisted  of  removing 
background  values  from  the  data,  converting  the  points  from  the  geographic  coordinate  system 
used  for  collection  to  the  Universal  Transverse  Mercator  Zone  13N  coordinate  system  used  for 
processing,  and  exporting  the  resulting  data  to  a  .CSV  file  that  could  be  imported  into  the  UX- 
Analyze  package  in  Geosoft’s  Oasis  Montaj  software.  The  exported  .CSV  file  name  contained 
both  the  collection  ID  and  the  Target  ID  (e.g.,  2621_Static00001_2621). 

6.2  PARAMETER  ESTIMATION 

All  MetalMapper  data  points  were  inverted  using  UX-Analyze  to  determine  modeled  parameters 
for  each  target.  These  parameters  included  the  location,  size,  and  orientation  of  the  source 
object;  the  polarizability  of  each  axis  of  the  object;  and  infonnation  regarding  the  quality  of  the 
data  and  the  relative  match  between  the  inverted  data  and  the  expected  model. 

All  target  inversion  was  initially  performed  using  the  UX-Analyze  batch  processing  mode  with 
the  multiple  object  solver  enabled.  Targets  for  which  multiple  objects  were  identified  using  the 
multiple  object  solver  were  re-inverted  using  the  batch  processor  without  the  multiple  object 
solver  enabled.  In  these  cases,  the  single  object  and  multiple  object  results  were  compared  to 
determine  which  method  returned  a  result  more  indicative  of  TOI.  Although  the  multiple  object 
result  may  have  approximated  the  expected  model  to  a  higher  degree,  the  result  more  indicative 
of  potential  TOI  was  used  for  target  ranking  to  be  conservative. 

6.3  CLASSIFIER  AND  TRAINING 
6.3.1  Confidence  Metrics 

The  polarization  curves  developed  for  each  target,  including  any  single-object-only  results  and 
secondary  multiple-object  results,  were  compared  to  a  library  of  known  polarization  curves 
compiled  using  test  stand  data  and  test  pit  data  from  the  former  PMTMA.  The  items  in  the 
comparison  library  were  limited  to  the  TOI  expected  at  the  PMTMA  site:  37mm,  75mm, 
105mm,  and  155mm  projectiles;  60mm,  81mm,  and  Stokes  mortars;  and  small  industry  standard 
objects  (ISOs).  Examples  of  various  types  of  these  items  were  used  (e.g.,  six  different  versions 
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of  37mm  projectiles,  three  types  of  81mm  mortar),  but  items  not  expected  at  the  site,  such  as 
hand  grenades  and  rockets,  were  not  included.  All  initial  comparisons  between  the  measured 
targets  and  the  library  data  were  also  performed  using  an  equal  weight  for  all  three  primary 
polarizabilities  (size:  1,  shape  1:  1,  shape  2:  1).  The  classification  results  for  each  target  were 
then  examined  by  the  data  processor. 

The  first  examination  of  the  classification  results  was  perfonned  to  detennine  the  usability  of 
each  result.  The  processor  either  detennined  that  the  results  were  usable  as  they  were  or  made  a 
note  in  the  target  database  in  Geosoft  that  further  processing  was  necessary.  Results  were 
deemed  usable  if  the  reviewer  identified  three  reasonable-looking  polarization  curves  or  if  a 
curve  for  the  primary  axis  of  polarization  ((31)  could  not  be  identified.  In  these  cases,  the  target 
was  either  left  for  ranking  according  to  the  decision  statistic  developed  for  the  project  (Section 
4.0)  or,  for  those  targets  without  an  identifiable  (31  curve,  classified  as  “can’t  analyze.”  While 
the  data  for  “can’t  analyze”  targets  were  not  usable  for  classification  purposes  because  UX- 
Analyze  cannot  effectively  compare  a  target  without  a  primary  polarizability  to  the  library  data, 
the  result  for  that  target  was  considered  “usable”  in  that  no  further  analysis  would  be  performed 
on  that  target. 

Targets  with  results  not  necessarily  deemed  usable  on  the  first  pass  included  those  for  which  one 
or  more  non-(31  curves  appeared  to  be  poor  data  for  any  reason  or  targets  that  appeared  to  be 
“ordnance-like”  but  did  not  have  a  particularly  good  match  to  any  of  the  library  objects. 
“Ordnance-like”  was  defined  as  an  object  with  relatively  equal  (i.e.,  symmetric)  secondary  axes 
of  polarizability  (P2  and  P3)  for  which  the  magnitude  of  pi  was  not  less  than  P2  and  P3.  It  was 
considered  possible  that  targets  with  these  characteristics  were  examples  of  ordnance  not 
expected  at  the  site  and,  therefore,  not  in  the  comparison  library.  Figure  6-1  shows  examples  of 
the  types  of  targets  flagged  during  the  first  examination  of  the  data:  one  with  poor  results  for 
multiple  polarization  curves,  and  one  that  appears  symmetric  but  with  a  poor  match  to  any 
library  object. 

Targets  with  one  or  more  apparently  poor  P2  or  P3  curves,  as  shown  in  Figure  6-1  A,  were  re¬ 
compared  to  the  library  data  with  the  poor  curves  removed  from  the  comparison.  This  was 
accomplished  by  changing  the  comparison  weight  for  the  poor-quality  data  to  0  (size:  1,  shape  1: 
1,  shape  2:  0;  or  size:  1,  shape  1:  0,  shape  2:  0).  In  these  cases,  the  revised,  (31/(32-  or  (31-only- 
based  confidence  metrics  were  used  when  calculating  the  decision  statistic  used  to  rank  each 
target. 


6.3.2  Training  Data 

The  training  data  request  for  the  former  PMTMA  comprised  targets  flagged  as  “ordnance-like,” 
as  described  above.  Various  examples  of  these  items  were  added  to  a  separate  target  library  and 
compared  to  the  other  symmetric  objects,  with  the  end  result  that  seven  items  added  to  the  library 
fit  all  of  the  others  with  a  confidence  of  0.75  or  higher.  Nine  examples  of  these  items  or  items 
closely  matching  them  were  requested  as  training  data.  None  of  the  requested  targets  were 
identified  as  a  TOI,  so  the  original  PMTMA  library  was  not  modified  based  on  the  results  of  the 
training  data. 
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Figure  6-1:  Targets  Flagged  for  Additional  Processing 


A:  P3  curve  appears  to  be  poor  data 
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6.3.3  Decision  Statistic 

Classification  for  the  PMTMA  project  was  accomplished  using  the  confidence  metrics  generated 
for  each  target  during  the  comparison  to  the  library  data.  Two  dig  lists  were  submitted  for  the 
project.  For  the  first,  the  confidence  metric  calculated  for  all  three  curves  was  used  unless  poor 
curves  were  identified  by  the  analyst,  as  described  in  Section  3.3.  In  that  event,  the  metric 
calculated  for  the  (31/(32-  or  pi-only  matches  was  used.  The  decision  statistic  used  for  the  project 
was  simply  the  final  confidence  metric  selected  for  each  target  (pl/p2/p3,  or  pl/p2-  or  pl-only). 
No  additional  weight  was  given  to  targets  that  had  three  usable  curves  rather  than  one-  or  two- 
only,  as  there  were  relatively  few  targets  for  which  less  than  three  curves  were  used.  Targets 
were  ranked  based  on  decreasing  confidence  that  they  were  TOI. 

For  the  second  list,  targets  were  prioritized  according  to  a  combination  of  pi/p2/p3,  pi/p2-,  and 
pi -only  confidence  metrics  prior  to  ranking.  Priorities  ranged  from  4  (high)  to  0  (low)  for  all 
non-training/non-can’t  analyze  targets,  and  the  decision  statistic  was  a  sum  of  the  priority  plus 
the  pi/p2/p3  confidence  metric.  Each  dig  list  was  completed  in  three  stages  as  described  below. 

6.4  DATAPRODUCTS 
6.4.1  Dig  List  1 

As  discussed  in  Section  6.3.3,  the  decision  statistic  for  the  first  list  was  based  solely  on  the  final 
confidence  metric  selected  for  each  target  regardless  of  the  number  of  polarization  curves  used  in 
the  match.  The  decision  statistic  was  equal  to  the  confidence  metric,  and  targets  were  ranked 
from  high  decision  statistic  to  low. 
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6.4.1. 1  Dig  List  1  Stage  1 

The  first  stage  dig  list  was  submitted  with  the  following  parameters: 

•  Training  Data:  9  items  selected  as  described  in  Section  3.4 

•  Can’t  Analyze:  164  items  without  usable  pi  curves 

•  Likely  TOI  (Category  1):  Decision  statistic  greater  than  0.700 

•  Cannot  Decide  (Category  2):  Decision  statistic  between  0.600  and  0.700 

•  Likely  Clutter  (Category  3):  Decision  statistic  less  than  0.600 

The  stop  dig  threshold  was  set  at  a  decision  statistic  of  0.650  (mid-Category  2),  with  the  training 
data  and  “can’t  analyze”  targets  also  considered  digs.  The  Stage  1  dig  list  was  only  compared  to 
the  project  seed  items  identified  as  QC  seeds.  This  comparison  identified  two  seeds  that  would 
have  gone  un-dug  based  on  the  Stage  1  list,  PM- 155  and  PM- 1847.  PM- 155  was  a  37mm  buried 
at  18cm  with  a  decision  statistic  of  0.608;  PM- 1847  was  a  small  ISO  buried  at  a  depth  of  30cm 
with  a  decision  statistic  of  0.632.  Because  both  of  these  targets  were  un-dug  targets  within 
Category  2,  the  decision  was  made  to  extend  the  dig  threshold  to  the  bottom  of  Category  2 
(0.600)  for  the  next  stage  of  the  dig  list. 

6.4.1.2  Dig  List  1  Stage  2 

The  second  stage  dig  list  was  submitted  with  the  following  parameters: 

•  Training  Data:  1 1  items,  including  QC  Seeds  PM- 155  and  PM- 1847 

•  Can’t  Analyze:  164  items  without  usable  pi  curves 

•  Likely  TOI  (Category  1):  Decision  statistic  greater  than  0.600; 

•  Cannot  Decide  (Category  2):  Decision  statistic  between  0.600  and  0.550 

•  Likely  Clutter  (Category  3):  Decision  statistic  less  than  0.550 

Modifications  between  the  Stage  1  dig  list  and  the  Stage  2  dig  list  included  the  addition  of  the 
two  missed  QC  seeds  to  the  training  data  set  and  the  changing  of  category  break  points  to  lower 
decision  statistics  based  on  the  non-detection  of  PM- 155  and  PM- 1847  with  the  Stage  1  dig  list. 
The  Stage  2  dig  list  was  considered  an  actual  dig  list,  and  the  full  set  of  intrusive  results  for  the 
targets  marked  as  “dig”  were  returned  following  the  submittal.  For  this  list,  the  stop  dig  point 
was  set  at  a  decision  statistic  of  0.575. 

The  dig  results  indicated  that  the  lowest  ranked  TOI  was  PM- 1492,  which  had  a  decision  statistic 
of  0.587.  The  final  19  investigated  targets  were  all  non-TOI,  so  there  did  not  appear  to  be  a 
reason  to  modify  the  dig  threshold  further  for  the  final,  Stage  3  dig  list. 

6.4.1.3  Dig  List  1  Stage  3 

The  final  dig  list  was  submitted  with  the  following  parameters: 

•  Training  Data:  1 1  items,  including  QC  Seeds  PM- 155  and  PM- 1847 

•  Can’t  Analyze:  164  items  without  usable  pi  curves 
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•  Likely  TOI  (Category  1):  Decision  statistic  greater  than  0.575  -  Dig 

•  Likely  Clutter  (Category  3):  Decision  statistic  less  than  0.575  -  Do  not  dig 

The  only  change  between  the  Stage  2  and  Stage  3  dig  lists  was  the  elimination  of  Category  2. 
Investigated  cannot  decide  targets  from  Stage  2  were  changed  to  Category  1,  and  targets  with 
decision  metrics  between  0.575  and  0.550  (i.e.,  un-dug  Category  2)  were  changed  to  Category  3. 
The  final  list  included  750  digs  out  of  the  2,370  total  targets.  One  hundred  and  seventy-five  of 
these  were  either  targets  used  as  training  data  or  “can’t  analyze”  anomalies,  with  the  remainder 
ranked  according  to  the  decision  statistic.  The  stop  dig  threshold  was  set  at  rank  number  575. 

6.4.2  Dig  List  2 

As  discussed  in  Section  6.3.3,  the  decision  statistic  for  the  second  target  list  submitted  for  the 
Pole  Mountain  project  was  based  on  a  combination  of  a  priority  given  to  each  target  and  the 
confidence  metric  for  that  target.  For  the  second  list,  targets  were  prioritized  according  to  a 
combination  of  pi/p2/p3,  f:S  1  /[12-,  and  pi-only  confidence  metrics  prior  to  ranking  as  follows: 

•  Priority  4  (highest  priority):  pl/p2/p3  confidence  metric  greater  than  0.80  and  pi-only 
confidence  metric  greater  than  0.85 

•  Priority  3:  Non-Priority  4  targets  with  pl/p2/p3  confidence  metric  greater  than  0.60  and 
pi -only  confidence  metric  greater  than  0.70 

•  Priority  2:  Non-Priority  3  or  4  targets  with  p l/p2/p3  confidence  metric  greater  than  0.575 
and  pi -only  confidence  metric  greater  than  0.70 

•  Priority  1  (lowest  priority):  Non-Priority  2,  3,  or  4  targets  with 

pl/p2/p3  confidence  metric  greater  than  0.575  and  pi-only  confidence  metric  less 
than  0.70 

pi/p2-only  confidence  greater  than  0.75  and  pi-only  confidence  metric  greater  than 
0.85,  provided  target  is  not  plate-like  (i.e.,  p2/p3  curves  are  greater  than  pi  curve); 
added  after  Stage  1  dig  list  (prior  to  Stage  2) 

The  decision  statistic  for  each  target  was  the  sum  of  the  priority  plus  the  pl/p2/p3  confidence 
metric  for  that  target. 

6.4.2. 1  Dig  List  2  Stage  1 

The  first  stage  dig  list  was  submitted  with  the  following  parameters: 

•  Training  Data:  9  items  selected  as  described  in  Section  3.4 

•  Can’t  Analyze:  167  items  without  usable  pi  curves 

•  Likely  TOI  (Category  1):  Decision  statistic  greater  than  3.0  (Priority  3  and  4  targets) 

•  Cannot  Decide  (Category  2):  Decision  statistic  between  1.0  and  3.0  (Priority  1  and  2 
targets) 

•  Likely  Clutter  (Category  3):  Decision  statistic  less  than  1.0 
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The  stop  dig  threshold  was  set  at  a  decision  statistic  of  2.0  (the  break  point  between  the  Priority  1 
and  Priority  2  targets  in  Category  2),  with  the  training  data  and  “can’t  analyze”  targets  also 
considered  digs.  The  Stage  1  dig  list  was  only  compared  to  the  project  seed  items  identified  as 
QC  seeds.  This  comparison  did  not  identify  any  missed  QC  seeds.  However,  one  of  the  TOIs 
was  ranked  extremely  close  to  the  stop  dig  threshold  on  this  list.  Therefore,  the  decision  was 
made  to  extend  the  dig  threshold  to  the  bottom  of  Category  2  (decision  statistic  greater  than  1) 
for  the  next  stage  of  the  dig  list.  Items  with  low  pi/(32/[33  but  relatively  high  (!2/[13-only  and  pi- 
only  confidence  matches  were  also  added  to  the  dig  list  as  Priority  1  targets. 

6.4.2.2  Dig  List  2  Stage  2 

The  second  stage  dig  list  was  submitted  with  the  following  parameters: 

•  Training  Data:  9  items  selected  as  described  in  Section  3.4 

•  Can’t  Analyze:  167  items  without  usable  pi  curves 

•  Likely  TOI  (Category  1):  Decision  statistic  greater  than  1.0  (all  non-0  Priority  targets) 

•  Likely  Clutter  (Category  3):  Decision  statistic  less  than  1.0 

As  discussed  above,  for  the  Stage  2  dig  list,  all  of  the  un-dug  Category  2  targets  were  re¬ 
classified  as  Category  1,  and  items  with  low  p l/p2/p3  but  relatively  high  p2/p3-only  and  pi-only 
confidence  matches  were  also  added  to  the  dig  list  as  Priority  1  targets  and  classified  as  Category 
1 .  Nine  targets  were  added  to  the  dig  category  based  on  their  re-classification. 

None  of  the  digs  added  between  the  Stage  1  and  Stage  2  dig  lists  were  TOI.  Therefore,  the 
decision  was  made  to  keep  the  Stage  2  dig  list  as  the  final  list.  Because  there  were  no  Category  2 
targets  in  the  Stage  2  list,  no  changes  were  made  for  Stage  3.  The  final  list  included  634  digs  out 
of  the  2,370  total  targets.  One  hundred  and  seventy-five  of  these  were  either  targets  used  as 
training  data  or  “can’t  analyze”  anomalies,  with  the  remainder  ranked  according  to  the  decision 
statistic.  The  stop  dig  threshold  was  set  at  rank  number  458. 
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7.0  PERFORMANCE  ASSESSMENT 


7.1  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOI 

IDA  compared  the  submitted  dig  list  was  compared  to  ground  truth  data  from  PMTMA.  The 
results  of  the  dig  list  comparisons  to  the  ground  truth  list  were  judged  according  to  performance 
objectives  identified  for  the  project  in  the  PMTMA  demonstration  plan  (Parsons,  2011). 
Table  3-1  contains  the  perfonnance  objectives  and  identifies  the  criteria  by  which  they  were 
judged.  The  results  for  the  two  submitted  dig  lists  with  respect  to  each  project  objectives  are 
detailed  below. 

Figure  7-1  shows  the  receiver  operating  characteristic  (ROC)  curve  for  dig  list  1.  As  indicated 
in  the  figure,  all  TOI  were  correctly  identified,  and  the  performance  objective  was  met. 


Figure  7-1:  ROC  Curve  for  Dig  List  1:  Confidence  Metric  as  Decision  Statistic 


PoleMtn  Parsons  UXAnalyze  None  MetalMapper  Custom  s2  vl  UXO 


No  Difficult  UXOs 


Figure  7-2  shows  the  ROC  curve  for  dig  list  2.  As  indicated  in  the  figure,  all  TOI  were  correctly 
identified,  and  the  performance  objective  was  met. 

7.2  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-TOI 

A  few  of  the  2,370  targets  for  which  data  were  collected  at  PMTMA  ended  up  being  multiple 
picks  on  the  same  source,  so  a  total  of  2,368  digs  were  performed  during  the  project.  This  also 
reduced  the  number  of  items  in  dig  list  1  that  needed  intrusive  investigation  from  753  to  749  and 
the  number  of  items  in  dig  list  2  from  634  to  633.  The  small  reduction  in  targets  meant  that 
there  were  2,208  true  negatives,  or  clutter  items,  in  the  data  set. 
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Figure  7-2:  ROC  Curve  for  Dig  List  2:  Prioritized  Targets 


PoleMtn  Parsons  UXAnalyze2  None  MetalMapper  Custom  s3  vl  UXO 
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Dig  list  1  correctly  identified  73.3  percent  (1,619  of  2,208)  of  the  clutter  items  as  clutter,  and  dig 
list  2  correctly  identified  78.5  percent  (1,735  of  2,208)  of  the  clutter  items  as  clutter.  Both 
versions  are  well  above  the  performance  objective  of  reducing  the  number  of  false  alarms  by 
greater  than  50  percent. 

7.3  OBJECTIVE:  CORRECT  SPECIFICATION  OF  NO-DIG  THRESHOLD 

For  both  dig  lists  submitted,  all  TOI  at  the  site  were  correctly  identified  as  TOI,  and  the  number 
of  false  positives  was  reduced  by  more  than  50  percent  of  the  total  number  of  false  positives. 
These  exceed  the  perfonnance  objectives  for  the  dig  threshold,  so  both  lists  have  passed  this 
performance  criterion. 

7.4  OBJECTIVE:  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE 
ANALYZED 

The  same  “can’t  analyze”  list,  which  contained  167  targets,  was  used  for  both  dig  lists  submitted. 
This  corresponds  to  approximately  7  percent  of  the  targets  at  the  site  and  exceeds  the  2  percent 
limit  specified  in  the  performance  objectives.  As  discussed  in  Section  3.3,  all  targets  without  a 
usable  pi  curve  were  classified  as  “can’t  analyze”  in  the  PMTMA  data  set.  It  is  expected  that 
additional  data  examination,  including  an  analysis  of  the  signal  amplitude  and  offset  between  the 
picked  and  collected  point  locations,  is  likely  to  reduce  the  number  of  “can’t  analyze”  targets 
required.  Using  this  strategy,  potential  “can’t  analyze”  targets  with  low  signal  amplitude 
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collected  where  they  should  have  been  collected  (i.e.,  likely  indicative  of  poor  EM61  data  for  the 
original  target)  would  be  considered  good  MetalMapper  data  despite  poor  pi  curves. 

7.5  OBJECTIVE:  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

The  target  parameters  estimated  in  this  case  were  the  X,  Y,  and  relative  Z  (depth)  coordinates  of 
the  targets.  Because  the  goal  with  this  objective  is  to  direct  the  dig  teams  to  the  correct  locations 
for  TOI,  the  comparison  of  estimated  coordinates  to  actual  coordinates  was  only  performed  for 
TOI  and  for  those  targets  marked  as  digs  in  the  ranked  dig  list.  Dig  list  2,  the  list  that  eliminated 
more  of  the  true  negatives  from  dig  consideration,  was  used  for  the  comparison. 

The  success  criteria  for  this  performance  objective  were  X,  Y  offsets  for  which  one  standard 
deviation  of  the  dataset  was  less  than  15  cm  and  depth  offsets  for  which  one  standard  deviation  of 
the  data  set  was  less  than  10cm.  The  performance  objective  was  passed  for  the  TOI  comparison, 
with  calculated  standard  deviations  of  1 1.1cm  for  the  horizontal  offset  and  6.5cm  for  the  vertical 
offset.  The  results  for  all  of  the  targets  marked  as  digs  were  above  the  criteria  for  both  the 
horizontal  and  vertical  offsets,  with  a  calculated  horizontal  offset  standard  deviation  of  33.7cm 
and  a  vertical  offset  standard  deviation  of  10.8cm. 

In  the  comparisons,  the  modeled  location  of  the  target  was  the  fit  X,  Y,  and  Z  coordinates  while 
the  actual  location  was  defined  as  the  closest  measured  location  to  the  fit  location.  This  only 
mattered  in  the  case  of  targets  for  which  multiple  objects  were  recovered  from  the  vicinity  of  the 
picked  location.  However,  much  of  the  discrepancy  seen  in  the  X,  Y  locations  for  all  of  the 
“dig”  targets  is  likely  due  to  multiple-object  locations  being  modeled  as  single  objects  during 
inversion.  There  does  not  seem  to  be  a  ready  solution  to  the  slight  discrepancy  between  the 
performance  objective  and  the  results  for  the  “dig”  targets,  although  the  results  for  the  TOI  were 
positive. 
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8.0  COST  ASSESSMENT 


8.1  COST  MODEL 

The  cost  model  for  the  former  PMTMA  demonstration  includes  the  cost  of  processing  and 
analysis  for  the  MetalMapper  data  collected  by  Sky.  Costs  were  broken  down  into  three 
categories:  processing,  analysis,  and  dig  list  compilation.  Processing  and  analysis  time  for  each 
task  was  measured  to  the  nearest  15-minute  interval. 


Table  8-1.  MetalMapper  Processing  and  Analysis  Costs 


Cost  Element 

Data  Tracked  during  Demonstration 

Costs 

MetalMapper  Data  Processing  and  Analysis 

$3.23/target 

Processing 

Time  required  to  perform  inversion  of  each 
target  using  UX- Analyze  and  to  create 
polarization  curve  figures  for  each 

1.5  min/target 

Analysis 

Time  required  to  examine  polarization 
curves  for  each  target,  identify  unknown 
items  for  training  data  request,  finalize 
confidence  metrics,  and  import  results  into 
Access  database 

1  min/target 

Dig  List  Compilation 

Time  required  to  organize  and  submit  two 
ranked  dig  lists 

0.2  min/target 

8.2  COST  DRIVERS 

Based  on  the  factors  described  above,  the  total  per-target  cost  for  the  processing  and  analysis  of 
the  MetalMapper  targets  collected  at  the  former  PMTMA  was  $3.23.  It  is  expected  that  this  cost 
is  on  the  low  end  of  what  could  be  expected  for  a  typical  classification  project,  based  on  the 
relative  ease  of  this  site  with  respect  to  the  types  of  ordnance  expected  at  the  site  (few  and  large) 
and  with  Parsons’  relative  inexperience  with  processing  and  analyzing  MetalMapper  data. 
Although  the  results  of  this  demonstration  were  positive  with  respect  to  the  performance 
objectives,  additional  experience  with  this  type  of  data  has  identified  additional  processing  and 
analysis  steps  that  may  have  eliminated  a  greater  number  of  clutter  items  from  the  dig  category 
on  both  ranked  dig  lists.  Examples  of  additional  processing/analysis  that  would  potentially 
improve  the  demonstration  results  include  the  calculation  and  analysis  of  signal  amplitude  and 
collection  location  versus  pick  location  to  reduce  the  number  of  “can’t  analyze”  targets,  as 
described  in  Section  7.4,  and  the  creation  and  use  of  parameter  space  plots  as  a  resource  to 
potentially  identify  targets  that  might  need  to  be  added  to  or  removed  from  the  dig  category 
based  on  size  and  decay  characteristics. 
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9.0  IMPLEMENTATION  ISSUES 


There  were  few  notable  implementation  issues  during  this  project.  No  large  issues  were 
identified  with  the  UX -Analyze  software  package,  and  project  objectives  were  achieved  for  the 
most  part.  The  most  notable  performance  objective  failure  was  the  relatively  large  percentage  of 
“can’t  analyze”  targets  identified  in  the  data  set.  During  the  project,  all  targets  with  poor  pi 
curves  were  classified  as  “can’t  analyze,”  resulting  in  a  “can’t  analyze”  list  that  included 
approximately  7  percent  of  the  targets  investigated.  It  is  likely  that  a  large  number  of  the  “can’t 
analyze”  targets  were  collected  at  EM6 1  target  locations  that  may  have  been  generated  as  targets 
due  to  EM61  noise  or  duplicate  pick  on  an  anomaly  better  represented  by  a  different  EM61 
target.  In  cases  such  as  these,  an  analysis  of  the  signal  amplitude  and  EM61  target  vs. 
MetalMapper  collection  vs.  MetalMapper  fit  locations  may  have  resulted  in  fewer  “can’t 
analyze”  targets. 
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Appendix  A:  Points  of  Contact 


POINT  OF 
CONTACT 

ORGANIZATION 

Phone 

Fax 

e-mail 

Role  in  Project 

Dr.  Jeff 

Marqusee 

ESTCP  Program  Office 

901  North  Stuart  Street, 

Suite  303 

Arlington,  VA  22203 

703-696-2120  (V) 

703-696-2114  (F) 
jeffrey.marqusee@osd.mil 

Director,  ESTCP 

Dr.  Anne 

Andrews 

ESTCP  Program  Office 

901  North  Stuart  Street, 

Suite  303 

Arlington,  VA  22203 

703-696-3826  (V) 

703-696-2114  (F) 
anne.andrews@osd.mil 

Deputy  Director, 
ESTCP 

Dr.  Herb  Nelson 

ESTCP  Program  Office 

901  North  Stuart  Street, 

Suite  303 

Arlington,  VA  22203 

703-696-8726  (V) 

703-696-2114  (F) 

202-215-4844  (C) 
herbert.nelson@osd.mil 

Program  Manager, 

MR 

Ms.  Katherine 
Kaye 

HydroGeoLogic,  Inc. 

11107  Sunset  Hills  Road, 
Suite  400 

Reston,  VA  20190 

410-884-4447  (V) 
kkaye@hgl.com 

Special  Projects 
Consultant,  MR 

Mr.  Daniel 

Ruedy 

HydroGeoLogic,  Inc. 

11107  Sunset  Hills  Road, 
Suite  400 

Reston,  VA  20190 

703-736-4531 

druedy@hgl.com 

Program  Assistant, 

MR 

Dr.  Shelley 
Cazares 

Institute  for  Defense 

Analyses 

4850  Mark  Center  Drive 
Alexandria,  VA  223 1 1 

703-845-6792  (V) 

703-578-2877  (F) 
scazares@ida.org 

Performance 

Assessment 

Ms.  Jane  Francis 

Wyoming  Dept,  of 
Environmental  Quality 

122  West  25  th  Street 
Herschler  Bldg.,  4-W 
Cheyenne,  WY  82002 

307-777-7092  (V) 
jfranc@wyo.gov 

State  Regulator 

Ms.  Adrienne 
Nunn 

Wyoming  Dept,  of 
Environmental  Quality 

122  West  25  th  Street 
Herschler  Bldg.,  4-W 
Cheyenne,  WY  82002 

307-777-6428  (V) 
anunn@wyo.gov 

State  Regulator 

A-l 


Mr.  Larry 
Sandoval 

Medicine  Bow-Routt  NF 

307-745-2337  (V) 
lwsandoval@fs.fed.us 

Laramie  District 
Ranger 

Mr.  Nate  Haynes 

Medicine  Bow  NF 

Laramie  Ranger  District 

2468  Jackson  St.,  Laramie, 
WY  82070 

307-745-2317 

nhaynes@fs.fed.us 

On  Site  Contact 

Ms.  Laura 
Percifield 

Project  Manager 
Environmental  Remediation 
Branch 

US  ACE  -  Omaha  District 

402-995-2761  (V) 
laura.j.percifield@usace.anny.mil 

USACE 

Representative 

Dr.  Stephen 
Billings 

Sky  Research,  Inc. 

445  Dead  Indian  Memorial 
Road 

Ashland,  OR  97520 

541-552-5185  (V) 

541-488-4606  (F) 
steve.billings@skyresearch.com 

Lead  PI 

Mr.  Heesoo 

Chung 

Sky  Research,  Inc. 

303-868-9309 

Heesoo.Chung@skyresearch.com 

Lead  Data  Collector 

Ms.  Victoria  A. 
Kantsios 

URS  Group,  Inc. 

2450  Crystal  Drive 

Suite  500 

Arlington,  VA  22202 

703-418-3030  (V) 

703-418-3040  (F) 

victoria  kantsios@urscorp.com 

Lead  PI 

Mr.  Darrell  Hall 

URS  Group,  Inc. 

402-334-8181 

darrell  hall@urscorp.com 

Project  Geophysicist 

Dr.  Dean 
Keiswetter 

SAIC 

120  Quade  Drive 

Cary,  NC  27513 

919-677-1560  (V) 

919-678-1508  (F) 
keiswetterd@saic  .com 

Lead  PI 

Mr.  Levi 

Kennedy 

Signal  Innovations  Group, 

Inc. 

1009  Slater  Rd„  Ste.  200 
Durham,  NC  27703 

919-323-3456  (V) 

919-287-2398  (F) 
lkennedy@siginnovations.com 

Lead  PI 

Dr.  Len  Pasion 

Sky  Research,  Inc. 

Suite  1 12A,  2386  East  Mall 
Vancouver,  BC  V6T  1Z3 

541-552-5185  (V) 

541-488-4606  (F) 
len.pasion@skyresearch.com 

Lead  PI 

A-2 


Mr.  Greg  Van 

Parsons 

1700  Broadway,  Suite  900 
Denver,  CO  80290 

303-764-1927  (V) 

303-831-8208  (F) 
Greg.Van@Parsons.com 

Lead  PI 

Mr.  John 

Baptiste 

Parsons 

1700  Broadway,  Suite  900 
Denver,  CO  80290 

303-764-8840 

John.E.Baptiste@parsons.com 

Data  Analyst 

Ms.  Elise  Goggin 

US  ACE,  Huntsville 

256-895-1635 

Elise.M.Goggin@usace.anny.mil 

Data  Analyst 

Mr.  Rick 
Grabowski 

USACE,  Omaha  District 

402-995-2284 

richard.j.grabowski@usace.army.mil 

Data  Analyst 

Mr.  Tim  Deignan 

Shaw  Environmental,  Inc. 

720-554-8273 

timothy .  deignan@shawgrp  .com 

Data  Analyst 

A-3 


